{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6cb0fb42-4770-4678-958f-eb8876d427a1",
   "metadata": {},
   "source": [
    "# Evaluate Text Generation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2741d78d-8de4-4373-9307-4f7076a46e92",
   "metadata": {},
   "source": [
    "| | |\n",
    "|----------|-------------|\n",
    "| Author(s)   | Renato Leite (renatoleite@), Egon Soares (egon@) |\n",
    "| Last updated | 10/22/2023 |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cef986b3-9680-4b63-8a3b-9d13b498984c",
   "metadata": {},
   "source": [
    "## BLEU"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a8166a0-c12c-418e-b20b-905e21d1c3fe",
   "metadata": {},
   "source": [
    "### Explanations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d217ffc-cd62-4e0d-a146-9e33e91146b1",
   "metadata": {},
   "source": [
    "#### Original paper"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fb6a983-1c6a-4f7e-905c-41c797a4d193",
   "metadata": {},
   "source": [
    "https://dl.acm.org/doi/pdf/10.3115/1073083.1073135\n",
    "\n",
    "$BLEU = \\text{Brevity Penalty}\\times(\\exp(\\sum_{n=1}^{N}w_n\\log(\\text{modified precision}(n))))$\n",
    "\n",
    "$N = 4$ - This is the baseline used in the paper\n",
    "\n",
    "$w_n = 1 / N$ - This is for using uniform weights\n",
    "\n",
    "\n",
    "$\\text{Brevity Penalty} =\n",
    "  \\begin{cases}\n",
    "    1       & \\quad \\text{if } c > r\\\\\n",
    "    e^{(1-r/c)}  & \\quad \\text{if } c \\leq r\n",
    "  \\end{cases}$\n",
    "\n",
    "\n",
    "\n",
    "$\\text{modified precision}(n) = \\cfrac{\\sum \\text{Count Clip}(n)}{\\sum \\text{Count n-gram}_{candidate}}$\n",
    "\n",
    "$\\text{Count Clip}(n) = min(\\text{Count n-gram}_{candidate}, max(\\text{Count n-gram}_{reference}))$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f537742b-f6dc-44cd-86f7-679c9d1f1dc3",
   "metadata": {},
   "source": [
    "#### Alternative explanation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d186cc4b-ba7f-49e5-87ae-c80d5fe91eeb",
   "metadata": {},
   "source": [
    "https://cloud.google.com/translate/automl/docs/evaluate#bleu\n",
    "\n",
    "$\\text{BLEU} = \\underbrace{\\vphantom{\\prod_i^4}\\min\\Big(1,\n",
    "       \\exp\\big(1-\\frac{reference_{length}}\n",
    "    {candidate_{length}}\\big)\\Big)}_{\\text{brevity penalty}}\n",
    " \\underbrace{\\Big(\\prod_{i=1}^{4}\n",
    "    precision_i\\Big)^{1/4}}_{\\text{n-gram overlap}}$\n",
    "\n",
    "$\\text{Brevity Penalty} = min(1, \\exp(1-\\cfrac{reference_{length}}{candidate_{length}}))$\n",
    "\n",
    "$\\text{n-gram overlap} = (\\displaystyle\\prod_{i=1}^{4} precision_i)^\\frac{1}{4}$\n",
    "\n",
    "$precision_i = \\dfrac{\\sum_{\\text{sentence}\\in\\text{Candidate-Corpus}}\\sum_{i\\in\\text{sentence}}\\min(m^i_{candidate}, m^i_{reference})}\n",
    " {w_{total Candidate}^i = \\sum_{\\text{sentence'}\\in\\text{Candidate-Corpus}}\\sum_{i'\\in\\text{snt'}} m^{i'}_{candidate}}$\n",
    " \n",
    "$m_{candidate}^i$: is the count of i-gram in the candidate matching the reference\n",
    "\n",
    "$m_{reference}^i$: is the count of i-gram in the reference\n",
    "\n",
    "$w_{totalCandidate}^i$:     is the total number of i-grams in the candidate"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25f9afca-4313-4fc4-b3f1-d8da1ca45db2",
   "metadata": {},
   "source": [
    "### Brevity Penalty"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ba9575c-9db6-44fb-aeea-268f153b627b",
   "metadata": {},
   "source": [
    "$\\text{Brevity Penalty} =\n",
    "  \\begin{cases}\n",
    "    1       & \\quad \\text{if } c \\geq r\\\\\n",
    "    e^{(1-r/c)}  & \\quad \\text{if } c < r\n",
    "  \\end{cases}$\n",
    "\n",
    "$ c = length_{candidate}$, $r = length_{reference}$\n",
    "\n",
    "$\\text{Brevity Penalty} = min(1, \\exp(1-\\cfrac{reference_{length}}{candidate_{length}}))$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "15a4a1d0-b8ba-416d-a25d-cc8f566c9500",
   "metadata": {},
   "outputs": [],
   "source": [
    "import math"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d1b959e0-01c5-4174-acb0-ac09ee7c119c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_brevity_penalty(reference_len: int, candidate_len: int) -> float:\n",
    "    # Raise an error if any number is negative\n",
    "    if reference_len < 0 or candidate_len < 0:\n",
    "        raise ValueError(\"Length cannot be negative\")\n",
    "    # If the candidate length is greater than the reference length, r/c < 1, exp(positive number) > 1,  brevity penalty = 1\n",
    "    if candidate_len > reference_len:\n",
    "        print(f\"Candidate length \\t ({candidate_len}) \\t is greater than the reference length \\t ({reference_len}), \\t so the Brevity Penalty is equal to \\t 1.000\")\n",
    "        return 1.0\n",
    "    # If the lengths are equal, then r/c = 1, and exp(0) = 1\n",
    "    if candidate_len == reference_len:\n",
    "        print(f\"Candidate length \\t ({candidate_len}) \\t is equal to the reference length \\t ({reference_len}), \\t so the Brevity Penalty is equal to \\t 1.000\")\n",
    "        return 1.0\n",
    "    # If candidate is empty, brevity penalty = 0, because r/0 -> inf and exp(-inf) -> 0\n",
    "    if candidate_len == 0:\n",
    "        print(f\"Candidate length \\t ({candidate_len}) \\t is equal to 0.0, \\t\\t\\t\\t so the Brevity Penalty is equal to \\t 0.000\")\n",
    "        return 0.0\n",
    "\n",
    "    # If the candidate length is less than the reference length, brevity penalty = exp(1-r/c)\n",
    "    print(f\"Candidate length \\t ({candidate_len}) \\t is less than the reference length \\t ({reference_len}),\\t so the Brevity Penalty is equal to \\t {math.exp(1 - reference_len / candidate_len):.3f}\")\n",
    "    return math.exp(1 - reference_len / candidate_len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ac9eb00a-f95d-440b-9e23-69d39f510542",
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_brevity_penalty_2(reference_len: int, candidate_len: int) -> float:\n",
    "    # Raise an error if any number is negative\n",
    "    if reference_len < 0 or candidate_len < 0:\n",
    "        raise ValueError(\"Length cannot be negative\")\n",
    "    # Avoid a division by 0\n",
    "    if candidate_len == 0:\n",
    "        if reference_len == 0:\n",
    "            return 1.0\n",
    "        else:\n",
    "            return 0.0 \n",
    "    return min(1.0, math.exp(1 - reference_len / (candidate_len)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "20dd3bd9-87f5-40dc-abf9-c785afcf29b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "candidates = [\"It is a guide to action which ensures that the military always obeys the commands of the party.\",\n",
    "              \"It is to insure the troops forever hearing the activity guidebook that party direct.\",\n",
    "              \"\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "583be0b0-5fca-405f-8745-61d8e20628b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "references = [\"It is a guide to action that ensures that the military will forever heed Party commands.\",\n",
    "              \"It is the guiding principle which guarantees the military forces always being under the command of the Party.\",\n",
    "              \"It is the practical guide for the army always to heed the directions of the party.\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "32abf1b6-06f9-4811-8a25-2fa2d7866ccb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from itertools import product"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a4119cd0-763c-47d3-b2fe-d7f3b73a6ad8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Candidate length \t (95) \t is greater than the reference length \t (88), \t so the Brevity Penalty is equal to \t 1.000\n",
      "Candidate length \t (84) \t is less than the reference length \t (88),\t so the Brevity Penalty is equal to \t 0.953\n",
      "Candidate length \t (0) \t is equal to 0.0, \t\t\t\t so the Brevity Penalty is equal to \t 0.000\n",
      "Candidate length \t (95) \t is less than the reference length \t (109),\t so the Brevity Penalty is equal to \t 0.863\n",
      "Candidate length \t (84) \t is less than the reference length \t (109),\t so the Brevity Penalty is equal to \t 0.743\n",
      "Candidate length \t (0) \t is equal to 0.0, \t\t\t\t so the Brevity Penalty is equal to \t 0.000\n",
      "Candidate length \t (95) \t is greater than the reference length \t (82), \t so the Brevity Penalty is equal to \t 1.000\n",
      "Candidate length \t (84) \t is greater than the reference length \t (82), \t so the Brevity Penalty is equal to \t 1.000\n",
      "Candidate length \t (0) \t is equal to 0.0, \t\t\t\t so the Brevity Penalty is equal to \t 0.000\n"
     ]
    }
   ],
   "source": [
    "bp1 = [calculate_brevity_penalty(len(reference), len(candidate)) for reference, candidate in product(references, candidates)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "32f475d4-ce7a-4651-8d02-61a02d57e0a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "bp_2 = [calculate_brevity_penalty_2(len(reference), len(candidate)) for reference, candidate in product(references, candidates)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8234bb98-b9bf-45fb-8c38-1bd89240c185",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bp1 == bp_2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49f12442-f24d-4179-a33d-3bf2c02274b4",
   "metadata": {},
   "source": [
    "### Precision"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cfcc65a-74e6-4073-ad96-7f91d4c6bfaa",
   "metadata": {},
   "source": [
    "$\\text{modified precision}(n) = \\cfrac{\\sum \\text{Count Clip}(n)}{\\sum \\text{Count n-gram}_{candidate}}$\n",
    "\n",
    "$\\text{Count Clip}(n) = min(\\text{Count n-gram}_{candidate}, max(\\text{Count n-gram}_{reference}))$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f23119a5-5254-4835-8aa4-0d9331db4854",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "from fractions import Fraction\n",
    "from itertools import tee\n",
    "\n",
    "\n",
    "def ngrams(sequence, n):\n",
    "    # Creates the sliding window, of n no. of items.\n",
    "    # `iterables` is a tuple of iterables where each iterable is a window of n items.\n",
    "    iterables = tee(iter(sequence), n)\n",
    "\n",
    "    for i, sub_iterable in enumerate(iterables):  # For each window,\n",
    "        for _ in range(i):  # iterate through every order of ngrams\n",
    "            next(sub_iterable, None)  # generate the ngrams within the window.\n",
    "    return zip(*iterables)  # Unpack and flattens the iterables.\n",
    "\n",
    "\n",
    "def count_clip(counts: Counter, max_counts: dict) -> dict:\n",
    "    clipped_counts = {}\n",
    "    for ngram, count in counts.items():\n",
    "        clipped_count = min(count, max_counts[ngram])\n",
    "        clipped_counts[ngram] = clipped_count\n",
    "\n",
    "    return clipped_counts\n",
    "        \n",
    "\n",
    "def calculate_modified_precision(references, candidate, n):\n",
    "    candidate = candidate.split()\n",
    "    candidate_counts = Counter(ngrams(candidate, n)) if len(candidate) >= n else Counter()\n",
    "    \n",
    "    max_counts = {}\n",
    "    for ref in references:\n",
    "        reference = ref.split()\n",
    "        reference_counts = (\n",
    "            Counter(ngrams(reference, n)) if len(reference) >= n else Counter()\n",
    "        )\n",
    "        for ngram in candidate_counts:\n",
    "            max_counts[ngram] = max(max_counts.get(ngram, 0), reference_counts[ngram])\n",
    "\n",
    "    clipped_counts = count_clip(candidate_counts, max_counts)\n",
    "    numerator = sum(clipped_counts.values())\n",
    "    \n",
    "    # Ensures that denominator is minimum 1 to avoid ZeroDivisionError.\n",
    "    denominator = max(1, sum(candidate_counts.values()))\n",
    "\n",
    "    return Fraction(numerator, denominator, _normalize=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b6689054-35d3-4e0b-94bb-1423c79532e2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "References\n",
      "\n",
      "It is a guide to action that ensures that the military will forever heed Party commands.\n",
      "It is the guiding principle which guarantees the military forces always being under the command of the Party.\n",
      "It is the practical guide for the army always to heed the directions of the party.\n"
     ]
    }
   ],
   "source": [
    "print(\"References\\n\")\n",
    "_ = [print(reference) for reference in references]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "854c433d-4ab5-4411-bf62-107bfdfc3f16",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Candidates\n",
      "\n",
      "Candidate 0 is 'It is a guide to action which ensures that the military always obeys the commands of the party.'\n",
      "Candidate 1 is 'It is to insure the troops forever hearing the activity guidebook that party direct.'\n",
      "Candidate 2 is ''\n"
     ]
    }
   ],
   "source": [
    "print(\"Candidates\\n\")\n",
    "_ = [print(f\"Candidate {i} is '{candidate}'\") for i, candidate in enumerate(candidates)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "66921fb0-cbbe-4217-9bf5-7572dd84732c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['The 1-gram modified precision for candidate 0 is 16/18',\n",
       " 'The 2-gram modified precision for candidate 0 is 10/17',\n",
       " 'The 3-gram modified precision for candidate 0 is 7/16',\n",
       " 'The 4-gram modified precision for candidate 0 is 4/15',\n",
       " 'The 1-gram modified precision for candidate 1 is 7/14',\n",
       " 'The 2-gram modified precision for candidate 1 is 1/13',\n",
       " 'The 3-gram modified precision for candidate 1 is 0/12',\n",
       " 'The 4-gram modified precision for candidate 1 is 0/11',\n",
       " 'The 1-gram modified precision for candidate 2 is 0',\n",
       " 'The 2-gram modified precision for candidate 2 is 0',\n",
       " 'The 3-gram modified precision for candidate 2 is 0',\n",
       " 'The 4-gram modified precision for candidate 2 is 0']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[f\"The {j+1}-gram modified precision for candidate {i} is {calculate_modified_precision(references, candidate, j+1)}\" for i, candidate in enumerate(candidates) for j in range(4)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b668a270-9c52-404c-84b7-8bbfda2ef7a9",
   "metadata": {},
   "source": [
    "### n-gram overlap"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35ee3d2c-c13b-413b-a751-73ec695092eb",
   "metadata": {},
   "source": [
    "$\\text{n-gram overlap} = \\exp(\\sum_{n=1}^{N}w_n\\log(\\text{modified precision}(n)))$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c12be6c4-0b4b-496c-b641-df5cf97d1d79",
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_n_gram_overlap(references, candidate, weights=(0.25, 0.25, 0.25, 0.25)):\n",
    "\n",
    "    # compute modified precision for 1-4 ngrams\n",
    "    modified_precision_numerators = Counter()  \n",
    "    modified_precision_denominators = Counter()  \n",
    "    candidate_lengths, reference_lengths = 0, 0\n",
    "\n",
    "    for i, _ in enumerate(weights, start=1):\n",
    "        modified_precision_i = calculate_modified_precision(references, candidate, i)\n",
    "        modified_precision_numerators[i] += modified_precision_i.numerator\n",
    "        modified_precision_denominators[i] += modified_precision_i.denominator\n",
    "\n",
    "    # remove zero precision\n",
    "    modified_precision_n = [\n",
    "        Fraction(modified_precision_numerators[i], modified_precision_denominators[i], \n",
    "        _normalize=False)\n",
    "        for i, _ in enumerate(weights, start=1)\n",
    "        if modified_precision_numerators[i] > 0\n",
    "    ]\n",
    "    weighted_precisions = (weight_i * math.log(precision_i) for weight_i, precision_i in zip(weights, modified_precision_n))\n",
    "    precisions_sum = math.fsum(weighted_precisions)\n",
    "\n",
    "    return math.exp(precisions_sum)\n",
    "\n",
    "def bleu(references, candidate, weights=(0.25, 0.25, 0.25, 0.25)):  \n",
    "    candidate_len = len(candidate.split())\n",
    "    references_lens = (len(reference.split()) for reference in references)\n",
    "\n",
    "    # Reference length closest to the candidate length\n",
    "    closest_reference_len = min(\n",
    "        references_lens, key=lambda reference_len: (abs(reference_len - candidate_len), reference_len)\n",
    "    )\n",
    "    brevity_penalty = calculate_brevity_penalty_2(closest_reference_len, candidate_len)\n",
    "    n_gram_overlap = calculate_n_gram_overlap(references, candidate, weights)\n",
    "    \n",
    "    return brevity_penalty * n_gram_overlap\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "687c273c-b699-4906-832b-00ecd6c3dd46",
   "metadata": {},
   "source": [
    "### BLEU"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3c8f1cd-2485-481c-be9b-ba698bd769ca",
   "metadata": {},
   "source": [
    "$BLEU = \\text{Brevity Penalty}\\times\\text{n-gram overlap}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "2740fbec-a85d-4829-9f72-478d448f2af4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def bleu(references, candidate, weights=(0.25, 0.25, 0.25, 0.25)):  \n",
    "    candidate_len = len(candidate.split())\n",
    "    references_lens = (len(reference.split()) for reference in references)\n",
    "\n",
    "    # Reference length closest to the candidate length\n",
    "    closest_reference_len = min(\n",
    "        references_lens, key=lambda reference_len: (abs(reference_len - candidate_len), reference_len)\n",
    "    )\n",
    "    brevity_penalty = calculate_brevity_penalty_2(closest_reference_len, candidate_len)\n",
    "    n_gram_overlap = calculate_n_gram_overlap(references, candidate, weights)\n",
    "    \n",
    "    return brevity_penalty * n_gram_overlap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "48ff8e0c-1aa9-4ab2-8ee0-7678e628d9e0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4969770530031034"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bleu(references, candidates[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "594ee9ed-e63e-470f-8615-3bd63ff9417e",
   "metadata": {},
   "source": [
    "### NLTK Implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "d3c7b0d1-90ad-456f-8055-ae26bbfb66a5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: nltk in ./venv/lib/python3.9/site-packages (3.8.1)\n",
      "Collecting nltk\n",
      "  Using cached nltk-3.8.1-py3-none-any.whl (1.5 MB)\n",
      "  Using cached nltk-3.8-py3-none-any.whl (1.5 MB)\n",
      "Requirement already satisfied: click in ./venv/lib/python3.9/site-packages (from nltk) (8.1.7)\n",
      "Requirement already satisfied: tqdm in ./venv/lib/python3.9/site-packages (from nltk) (4.66.1)\n",
      "Requirement already satisfied: joblib in ./venv/lib/python3.9/site-packages (from nltk) (1.3.2)\n",
      "Requirement already satisfied: regex>=2021.8.3 in ./venv/lib/python3.9/site-packages (from nltk) (2023.8.8)\n"
     ]
    }
   ],
   "source": [
    "!pip install -U nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "0120c708-6a02-41c6-ab9d-fbe107639788",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4969770530031034\n"
     ]
    }
   ],
   "source": [
    "from nltk.translate.bleu_score import sentence_bleu\n",
    "\n",
    "nltk_bleu_score = sentence_bleu([reference.split() for reference in references], candidates[0].split())\n",
    "print(nltk_bleu_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71027184-0c5a-46ab-a345-d02f0349fdb6",
   "metadata": {},
   "source": [
    "## ROUGE-L"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c3dd86f-daae-4a27-924c-26be4d0cb9dd",
   "metadata": {},
   "source": [
    "See Theory_Evaluate_2_Summarization.ipynb"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
